Determining Alertness in Individuals with Profound Intellectual and Multiple Disabilities: The Reliability of an Observation List

In the support of individuals with profound intellectual and multiple disabilities (PIMD), assessing the level of alertness is a recurring issue for parents and other direct support persons. Although observations show clear advantages above and beyond other assessment methods, there are problems related to this method as well. Subjectivity of interpretation and low reliability results have been described as the main problems. In the present study, our aim was to estimate the reliability of the Alertness Observation List (AOL) while, at the same time, minimizing the problems entailed in observations. We calculated both the inter-observer agreement and intra-observer agreement for 39 situations. Since the results exceeded the formulated 80%-criterion, we concluded that the AOL was a reliable instrument. However, the large range found in the results was striking. Moreover, observers with different information about the observed individuals came up with different reliability scores. To determine the value of observation of individuals with PIMD, it might well be necessary to judge the actual usefulness that the instrument has in clinical practice, besides the reliability of the results. While stimulation to promote communication and learning is essential for the support of individuals with profound intellectual and multiple disabilities (PIMD) (Guess et al., 1993), direct support persons (DSPs) regularly wonder how to determine the “right moment” for starting such stimulating activities. By the same token, it is important for an activity to be started at the “right moment” so as to allow time for the stimuli that are presented to enter the consciousness of the individual with PIMD (Nelson, van Dijk, McDonnell, & Thompson, 2002). The “right moment” has also been described as “being focused on the environment” or as “being alert” (Munde, Vlaskamp, Ruijssenaars, & Nakken, 2009). The questions that arise, then, refer to a number of topics: How does an individual show that he or she is focused on the environment? How can individual differences in alertness signals be interpreted? Can we determine an optimal moment during the day for stimulation of an individual with PIMD? There is an additional problem in that research shows that reduced levels of alertness and quick, irregular changes in alertness levels over time are common for individuals in the target group (Guess, Roberts, & Guy, 1999). These factors may even aggravate the problem of determining alertness reliably in individuals with PIMD. Despite agreement about the importance of determining alertness for the support of individuals in the target group, it is not obvious how different alertness levels ought to be determined. Since individuals with PIMD do not express their needs by means of spoken language, self-report cannot be used (Vlaskamp, 2005). Similarly, physiological measurements often show unusual patterns and do not reveal the necessary information about the complex behavior of individuals in the target group (Mudford, Hogg, & Roberts, 1997). In contrast, most authors do agree that alertness can be described in terms of observable behavior. Consequently, most instruments used to investigate and determine alertness in individuals We would like to thank the children and teachers who agreed to participate in this study. Special thanks are owed to the observers for their time and effort. Correspondence concerning this article should be addressed to Vera Munde, Department of Special Education, University of Groningen, Grote Rozenstraat 38, 9712 TJ Groningen, the NETHERLANDS. E-mail: [email protected] Education and Training in Autism and Developmental Disabilities, 2011, 46(1), 116–123 © Division on Autism and Developmental Disabilities 116 / Education and Training in Autism and Developmental Disabilities-March 2011 with PIMD are based on observations (Munde et al., 2009). Observations clearly have a number of advantages above and beyond other assessment methods for individuals with PIMD. As a consequence of the severity of their disabilities, individuals with PIMD are not able to use spoken language, and so they express themselves by means of body language. Consequently, individuals in the target group often cannot fulfill the requirements of standardized assessment instruments in terms of motor and speech abilities (Vlaskamp, 2005). The communication of individuals with PIMD mostly consists of subtle signals that are difficult to detect for DSPs (Wilder & Granlund, 2003). The same signal may have a different meaning for different individuals (Vlaskamp). While physiological measurements can help to register these subtle signals, the results do not reveal the necessary information about the meaning of these signals for individuals in the target group (Mudford et al., 1997). Looking at the individual’s reactions in different situations, DSPs can learn to interpret the different kinds of behavior (Grove, Bunning, Porter, & Olsson, 1999). Only observations allow DSPs to take the meaning of the individual’s behavior into account. Detailed registration of the behavior and, at the same time, of the influencing factors are especially important when observing individuals with PIMD. However, general problems are related to observations in individuals with PIMD as well. Observations often lack an unambiguous description of their focus. When observations are based on theoretical concepts, these cannot be directly linked to visible behavior. Consequently, observers are forced to interpret the visible behavior, and ascribing meaning to behavior is, in turn, always interpretation (Vlaskamp, 2005). Interpretation, then, can be specified as yet another problem to do with observations. While it is important for DSPs to take the meaning of the behavior of their clients into account, several factors can bias the interpretation. DSPs interpret the behavior of the individual with PIMD based on their knowledge of the individual and previous experiences with the individual in similar situations. Since this knowledge differs for each DSP, observations of the same situation may result in different scores (Grove et al., 1999). Additionally, general expectations of reactions and contextual factors in a specific situation can also influence the DSP’s judgment (Hogg, Reeves, Roberts, & Mudford, 2001). While DSPs’ overall judgments remain similar in situations with and without contextual information, DSPs judge the individual’s expressions more positively when they expect the individual with PIMD to enjoy an activity than when they do not know about the content of the activity. Furthermore, and as a consequence of the subjectivity of the interpretations, observations of individuals with PIMD regularly result in low reliability. As a result, researchers find themselves still involved in discussions about influencing factors and explanations (Vlaskamp). The general problems that we experience in observations of individuals with PIMD also become apparent in the alertness observations of individuals in the target group. In the literature, no unambiguous description of alertness has been found (Munde et al., 2009). Although the authors all agreed that it was possible to observe alertness in the behavior of individuals with PIMD, different terms with different descriptions were found to have been introduced. Additionally, different scoring categories were used to determine alertness levels. Another point of discussion is scoring frequency. Because of quick and irregular changes in alertness levels, some authors plead the case for continuous scoring (Guess et al., 1999; Mudford et al., 1997). However, the difference in content information based on interval scoring is not yet made evident here, and, above all else, it should be remembered that interval scoring is actually more useful in clinical practice. To measure and compare the impact of these differences, researchers found themselves obliged to determine the reliability of their observations. For a number of the studies, reliability did not exceed the formulated criterion (Mudford et al.; Woodyatt, Marinac, Darnell, Sigafoos, & Halle, 2004). Although different explanations for these results have been discussed (Arthur, 2000; Guess, Roberts, Behrens, & Rues, 1998; Mudford, Hogg, & Roberts, 1999), no solution for the problem of low reliability in alertness observations has been offered. Taking the above-mentioned problems into account, the Alertness Observation List (AOL) Determining Alertness / 117 has been developed accordingly (Vlaskamp, Fonteine, & Tadema, 2005). Within the AOL, a clear description of alertness is employed. Alertness is described as the “level” of being open to the environment. DSPs are thus able to use the AOL to formulate an individual alertness profile. Alertness is scored on four different “levels” in order to search for alertness patterns over the period of a day and to find out about changes in alertness based on the impact of different stimuli. Thereby, scoring frequency increases for each of the three subsequent scoring forms of the AOL. The aim of the present study was to estimate the reliability of the AOL. In a previous study, the AOL was proved to be reliable in five cases (Petitiaux, Elsinga, Cuppen-Fonteine, & Vlaskamp, 2006). In the present study, we determined the general reliability of the instrument for a larger sample. In doing so, we strived to reach adequate reliability results while, at the same time, minimizing the problems with observations mentioned above.